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ABSTRACT 



The EuroMET (European Meteorological Education and Training) 
project was created to address the education and training needs of 
professional meteorologists and students in tertiary education throughout 
Europe and more widely. Two courses/ each modular in format, have been 
developed for the World Wide Web. The courses have been evaluated at three 
key points during their development for their usability, scientific content, 
and perceived pedagogical worth. By performing the evaluation in this way, 
recommendations for improvement could be made to the developers, the 
improvements incorporated, and all modifications fully tested before the 
course was considered ready for use. The final versions of the courses are 
currently undergoing a summative evaluation as they are used in real teaching 
and training environments. This paper describes the evaluation methods used 
during the project and the results obtained thus far. A table presents data 
on improvement in test scores before and after users complete a module. 
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Abstract: EuroMET has been created to address the education and training needs of 
professional meteorologists and students in tertiary education throughout Europe and more 
widely. Two courses, each modular in format, have been developed for the WWW. The 
courses have been evaluated at three key points during their development for their usability, 
scientific content and perceived pedagogical worth. By performing the evaluation in this 
way, recommendations for improvement could be made to the developers, these 
improvements incorporated, and all modifications fully tested before the course was 
considered ready for use. The final versions of the courses are currently undergoing a 
summative evaluation as they are used in real teaching and training environments. This 
paper describes the evaluation methods used during this project and the results obtained thus 
far. 
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Introduction 

The EuroMET (European Meteorological Education and Training) project arose due to both the emerging 
interest within the meteorological community for Computer Aided Learning and the decision of the European 
Commission to foster telematic applications within the 4 th programme for research and development. The 
education and training of meteorologists varies widely across Europe. Basic meteorology is taught in many 
universities, most often as part of other courses, for example geography or environmental science. In 
universities where it is offered as a comprehensive course, meteorology is generally taught in specialist 
departments at both undergraduate and postgraduate level. Due to the relatively small number of universities 
teaching meteorology at sufficiently advanced levels, National Meteorological Services (NMS) usually recruit 
employees at university degree level and then offer their own training as is necessary or possible. The NMS' 
education and training courses that exist at present differ greatly across Europe according to the size and 
organisation of each service. For example, some NMS run their own training courses, such as in France and the 
UK, and also offer these to other countries unable to offer their own, eg the Netherlands, while other NMS offer 
very little formal training, eg in Belgium. An important part of these training programmes is extension 
training, required because meteorological science and technology is continually developing so that new 
methods and theories need to be taught. At present no distance or open learning courses in meteorology exist 
within Europe. 

Thus it can be seen that within the European meteorological community there is a need for computer-based 
learning resources not only to enable universities and NMS* training centres to share material, but also to offer 
an alternative to those NMS which either send staff on costly and time-consuming courses, or currently offer 
little or no training or education. EuroMET has therefore been created to meet this need. It aims to address the 
education and training requirements of professional meteorologists and students in tertiary education 
throughout Europe by establishing a multimedia, networked-based, open and distance learning service. 
Moreover, this service will be provided in such a way that it can be easily customised to fit local requirements. 
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The EuroMET consortium is comprised of 23 meteorological partners, i.e. universities and National 
Meteorological Services (NMS) from 15 European countries and the University of Quebec. The consortium is 
divided into two groups, the developers who are responsible for delivering the course, and the evaluators who 
are responsible for assessing the course. The developers are led by M6t6o France, where the project co-ordinator 
is also based, and the University of Edinburgh. The evaluation is lead by EUMETSAT (the European 
Organisation for the Exploitation of Meteorological Satellites) and the University of Reading. The 30 month 
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project commenced in June 1996 and is split into three key stages. The first stage, lasting 6 months, was spent 
producing prototypes. The second stage lasted 12 months, during which time modules for the course were 
written by the developers, assessed by the evaluation team, and then returned to the developers for modification 
if necessary. The final stage, the demonstration phase, began in December 1997 and will run for a year, during 
which time the course will be used in universities and NMS on existing courses and as stand-alone education 
and training material. 



The Course 

It was decided that the course would be web-based, to make it easily accessible to everyone within the 
consortium, easy to include new developments and up-to-date material and also to make it possible to run 
powerful simulation models. The meteorological subjects for the course were selected in response to a 
European-wide needs analysis of National Meteorological Services. The two chosen were Numerical Weather 
Prediction (NWP) and Satellite Meteorology (SM). Both courses are comprised of nine chapters covering 
different sub-topics with each chapter containing a number of modules and totalling around 70 modules for 
each course. This modular approach allows teachers at different locations to choose only those modules which 
are directly relevant to the course they teach. 

A common interface was chosen so that each module would look and feel the same. The interface was designed 
to be as simple as possible so that users could learn to use it as they went along. There is however an online 
help facility which explains all the navigation buttons and symbols and icons used throughout the course. It was 
also designed to be language independent, an important consideration for a multilingual project. Each module 
was constructed in the same way, opening with a motivation page to introduce the topic and engage the user's 
interest. The next page lists the learning objectives of the module, any pre-requisite modules the user should 
study first, any subsequent modules which follow on, and a list of module contents. The following pages then 
contain the teaching material, whilst the final page contains a summary of the material, which draws out the 
salient points of the topic. Users navigate through the modules by either clicking on arrows on a button bar at 
the bottom of the screen to progress page by page, either forwards or backwards, or they can click on the 
’module structure’ button, which enables the user to go directly to any page of their choice. A ’course structure’ 
button similarly allows the user to select any module. 

In support of this learning environment is a glossary to explain commonly-used meteorological terms and a 
keyword index which lists modules which contain material on important terms. Facilities for one-to-one and 
one-to-many communication are also included, such as email to subject experts, discussion groups and internet 
relay chat (IRC) areas. Tools to create adapted packages and extra material are also available from the 
EuroMET WWW site with full documentation, to allow teachers to create an online course from the EuroMET 
modules completely tailored for their needs. 



Course Evaluation 

The aim of the evaluation is to assess the course in terms of its ease of use, pedagogical effectiveness, including 
ensuring its scientific integrity, and utility in replacing conventional teaching, with emphasis on the 
multimedia aspects of the modules. The course is being evaluated in two main phases, one phase ran during the 
development work and the other after the development work was completed. The EuroMET evaluation scheme 
has been planned and executed by all the evaluation partners with occasional advice from staff at the Institute 
of Educational Technology, Open University, UK and from the EC-sponsored BASELINE and INUSE centres. 



User Evaluation 

The user, or formative, evaluation lasted for 8 months and ran concurrently with the development phase, in this 



way the developers could incorporate any changes that were considered necessary before the next evaluation 
phase began. The modules were released on to the WWW in three batches, after each module release the 
evaluators had 4-6 weeks in which to make their assessments and report their findings back to the developers. 
The modules were tested for their ease of use and pedagogical effectiveness with a view to modifying them, if 
necessary, so that they meet the learners’ needs. 

The user evaluation had two components. The first component was a quality check of every module, performed 
at least once by every partner, and was concerned primarily with the overall look and feel of each module and 
its ease of use. The second component was a more detailed look at the modules, with each partner being 
assigned a small number of modules to be assessed by several people within that institution, preferably covering 
a wide range in meteorological background and experience, such that each module was tested in at least four 
different institutions. This more detailed examination included checking the scientific integrity, determining 
users’ impressions of different module characteristics, the level of interactivity, whether the module fulfilled its 
stated objectives, obtaining any suggestions of corrective action or possible improvements for the developers, 
and wherever possible testing the user’s understanding before and after using the module. Two questionnaire 
forms were designed to fulfil these criteria, these forms were in spreadsheet format and were distributed and 
collected electronically. 



Demonstration Phase 

The second evaluation phase is the demonstration, or summative, phase. This phase runs for one year, 
commencing 1 December 1997. The aim of this part of the evaluation is to assess the modules in use on 
teaching and training courses in a variety of learning environments, and to assess their effectiveness in 
comparison to existing teaching methods. All partners, developers and evaluators, will participate in this phase. 
As the consortium includes partners from organisations of different sizes with different needs, as well as 
different facilities and resources available, the flexibility of the course can be fully tested. 

Universities and some weather services can, for example, use the modules in conjunction with existing lecture 
courses. The modules can be incorporated in various ways ranging from the modules being used as 
complementary material which students are encouraged to look at in their own time, to full lecture replacement 
with dedicated classroom computer sessions. NMS which do not have their own training centres may well have 
to address the challenge of establishing how the modules can be used for ‘on-the-job’ training in an operational 
weather forecast environment. Potential use of the modules for these partners includes: giving individuals or 
groups dedicated time during their duty period for distance learning with the modules, supported by either a 
virtual tutor or a tutor group that meets at regular intervals; attempting to integrate the modules into the work 
schedule so that an individual could, for example, use a set of modules related to a weather-producing 
phenomenon and then look for real examples of the phenomenon while working, and using the modules as 
'electronic job aids' that can be referred to as and when necessary during the course of the working day. 

The pedagogical effectiveness of the EuroMET course will be assessed in various ways. For example, there is 
some scope during this project for the same course at an individual institution to be run in successive years with 
and without using the modules. Another option is to divide a student class into groups with perhaps one group 
using modules only, one having lectures and using modules, and a third group having lectures only. 
Examination or test marks could then be compared as a measure of the course's effectiveness. However, this is 
not always desirable when students' qualifications are at stake, particularly if some students feel they are at a 
disadvantage. Further evaluation of this sort can also be undertaken by higher degree students. Sometimes these 
students are required to take taught courses early on in their research programme, and so they could be used as 
assessors. Not all NMS examine their employees so evaluation of the modules’ pedagogical effectiveness could 
take the form of users giving seminars to their colleagues. In some cases they may be given forecasting 
exercises to test their understanding of the material covered in the modules. These assessment methods could 
also be used in universities. 

It is also important to gather information on the users' and teachers’ opinions and feelings about the course. 
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This information will come from a range of questionnaires available online. The main questionnaires are the 
’user’ and ’teacher’ questionnaires which will be completed by everyone using the course, either for learning or 
teaching. They address such issues as how the modules were used, if users received enough support or feedback 
whilst using the modules, how easy they were to incorporate into their studies/workload, as well as gathering 
general views on the comparative value of the modules and whether users felt they had learnt from them. The 
teacher questionnaire asks for views on the comparative value of the modules also, and about the impact it had 
on them, and provides areas to describe any student assessments, observations or interviews conducted. Both 
questionnaires also pose questions regarding the usage and usefulness of the communication facilities available 
with the course. 



Evaluation Results 

Initial results from the user evaluation phase have been collected and analysed. An independent usability study 
has also been carried out with the EC-supported project INUSE (Information Engineering Usability Support 
Centres), a network of centres specialising in usability issues. 



User Evaluation Results 

The first user evaluation phase found a disparity in the modules, with some placing the emphasis on using 
interactive and graphic elements to explain meteorological concepts, while others stuck to a more traditional, 
text-based approach. Users generally did not like the very heavy use of text, but preferred the use of animation 
and interactive exercises, as they offer a unique way of visualising concepts. It was felt therefore that the 
developers should have made more use of the WWW’s potential. The navigation tools also caused problems, 
with users often unsure if they had accessed the whole of the module or feeling that they were ‘lost’ in the 
module as occasionally there appeared to be no clear route through it. A further problem was the definition of 
the objectives of each module. These were sometimes poorly defined in terms of learning goals and very often 
just resembled the table of contents, making it very hard for a user to decide whether they had successfully 
completed a module. 

There were also some very good features present in some modules and which were recommended to be included 
in all modules. These features included the ‘sensitive’ equations and the conclusion pages at the end of a 
module. Sensitive equations were particularly useful when a module contained complex mathematical 
expressions with many terms which made it difficult to comprehend. Making each term sensitive, so that when 
the cursor was positioned over a term a pop-up explanation appeared on the screen, meant users found 
expressions far easier to understand. The conclusion page was also useful as it summarized the material and 
concepts covered and reaffirmed the important points. These features were incorporated into the Phase II 
modules. Further improvements included the use of more interactivity, though the evaluators felt that some 
modules were still too dry and formal, and the introduction of a timer bar which showed not only the time the 
user had been in a particular module, but also the fraction of it accessed. Navigation was also improved to guide 
the user through the module in a more linear manner to avoid being or feeling lost. 

The Phase III evaluation showed that there was an overall improvement in quality of the modules since the 
Phase II evaluation. The amount of interactivity in the modules improved with most considered to have 
sufficient interactivity. There was also a more imaginative use of interactivity with a wider variety of exercises 
for the user to perform. The objectives were much better defined, and the navigation had further improved with 
the introduction of ’course structure’ and ’module structure’ windows which close automatically once a module 
or page has been selected. This is a big improvement when navigating round the course. The only potential 
problem identified is that of variation in material between modules, that is the amount of material in a module 
and its relative difficulty. Users might become frustrated if, after long exploration of a module, they find it does 
not meet their expectations because it is either too difficult or too easy. However, this problem has not been 
tackled and, as the majority of modules have been evaluated out of sequence in this phase, so that sometimes it 



has been necessary to test advanced modules without first studying its preceding modules, it remains to be seen 
whether this is a real problem or is simply an artefact of the evaluation procedure thus far. 

Usability Study Results 

A usability study was conducted at Reading University in collaboration with INUSE, with representative users 
using the modules to perform typical tasks in a realistic environment. A representative from an INUSE centre, 
NPLUS (National Physical Laboratory Usability Services) in Teddington, UK, supervised the study. 

The usability study concluded that the course modules were well-designed and meet the needs of the users. 
Module presentation was good, with the right amount of information on each screen, the graphics and 
interactive elements were well liked, and the download times were acceptable. Furthermore, the results showed 
that it was likely that teaching staff will be able to successfully integrate modules into their courses and that the 
EuroMET course will provide a useful and usable study aid. Minor technical problems occurred during the 
study, such as graphics not loading correctly and windows that did not close when they should, but these have 
since been corrected by the developers. A recommendation for the quizzes within the modules was that users 
expressed a desire for some to be designed where they were required to submit answers of their own rather than 
merely selecting from a list of possible answers in order to fully test their understanding, this was also 
addressed in the Phase III modules. 



Pre- and Post-test Results 

Pre- and post-tests have been carried out only on a small scale during the user evaluation due to a number of 
reasons. One difficulty was with short time-scales, particularly for the first and third phases. Modules were 
delivered onto the WWW at the beginning of each evaluation phase, so in order to design pre- and post-tests, 
the evaluation partners had to read through each module and devise questions before the users could access the 
modules. This was a lengthy process and with only a few weeks for each evaluation period it simply was not 
possible for every institute to do this. A further problem was finding representative users to do the tests. As part 
of the user evaluation both subject experts and novices tested the modules. Subject experts are essential to check 
that the material in each module is scientifically correct, while novices can test the pedagogical effectiveness. 
However, due to the tight project timetable, the first two user evaluations occurred out of term time for most 
universities which have by far the largest proportion of learners. Added to this is the fact that the modules were 
not delivered in sequence. Some of the more complex modules require users to have covered others in the 
course first therefore a newcomer may have problems understanding material in a particular module, not 
because the module is poor, but because they do not have sufficient background knowledge. 
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Table 1: Improvement in test scores before and after users complete a module. 



Despite these problems, the results from the modules for which pre- and post-tests have been undertaken are 
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encouraging as they suggest that users are able to understand and retain information which helps them improve 
their level of knowledge (Table 1). They also provided the evaluation team with valuable experience for the 
demonstration phase when much more extensive tests will be possible as many representative users will be 
using the modules as part of a structured education or training course. 

The Future of EuroMET 

The demonstration phase provides an excellent opportunity to fully test the pedagogical effectiveness of the 
course and its utility in replacing and complementing existing education and training methods. A meeting to be 
held part-way through this phase, at the end of March 1998, will ensure that the evaluation tools are well- 
defined and appropriate for all partners, and may hopefully yield some preliminary results on the course’s use 
and effectiveness. 

It is hoped that after the demonstration phase institutions will continue to use the EuroMET course. 
Maintenance of the course servers for a minimum period of 2 years is guaranteed. Interest in the course has also 
been expressed by individuals and organisations who are not in the consortium but who have joined the 
EuroMET user group. It is expected that these members will be allowed access to the course at the end of the 
project in return for some level of commitment to the consortium, either financial or in terms of developing or 
evaluating new material. 

If this whole exercise is seen to be successful by the Informal Conference of Western European Directors 
(ICWED) of Weather Services it is hoped that further development of similar modules will be funded through 
their collaborative ‘EUMETNET’ venture. 
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